Image processing device, image processing method, and program

ABSTRACT

A device and a method are provided which perform movement vector calculation processing from photographing images for which a microlens array is used without generating whole images. Specifically, consecutive photographing images photographed by an imaging section having a microlens array are input, and a movement vector between the images is calculated. A movement vector calculating section calculates the movement vector corresponding to a feature point by using microlens photographing images photographed by respective microlenses. When the movement vector corresponding to the feature point within duplicate image regions having the same image region within a plurality of microlens photographing images is calculated, an average value of a plurality of movement vectors corresponding to the same feature point is calculated and is set as the movement vector of the feature point. Alternatively, an average value of plural movement vectors remaining after an outlier (abnormal value) is excluded is calculated and is set as the movement vector of the feature point.

TECHNICAL FIELD

The present disclosure relates to an image processing device, an imageprocessing method, and a program. The present disclosure particularlyrelates to an image processing device, an image processing method, and aprogram that detect a movement vector (optical flow) from photographingimages of an imaging device using a microlens array (MLA: Micro LensArray).

BACKGROUND ART

An imaging device using a microlens array (MLA: Micro Lens Array) inwhich a plurality of microlenses is arranged is known. A distancebetween a lens and an imaging element (image sensor) can be reduced byusing the microlens array (MLA), so that a thin imaging device can beconfigured.

The imaging device can be mounted in a thin apparatus such as asmartphone and is estimated to be used widely in the future.Incidentally, PTL 1 (Japanese Patent Laid-Open No. 2010-240215), forexample, describes an imaging device using a microlens array (MLA).

There is a fingerprint authenticating device of an electronic apparatussuch as a smartphone, as a concrete example of usage of an imagingdevice using a microlens array (MLA).

A sequence of fingerprint authentication processing is, for example, thefollowing processing. First, a moving image of a finger of a user isphotographed. Next, alignment of a plurality of consecutivephotographing images included in the moving image is performed, and theimages after the alignment are synthesized to generate a high qualityimage. Finally, processing of verifying the generated high quality imageagainst registered fingerprint data registered in a memory in advance isperformed.

The synthesis of the plural images is performed because, for example,only one still image has insufficient characteristic information of thefinger and thus decreases authentication accuracy. The synthesis of theplurality of consecutive photographing images needs the alignment of theplurality of consecutive photographing image frames. This alignmentneeds the processing of calculating the movement vector of a featurepoint included in the images.

However, in the microlens array (MLA) imaging device, an imagephotographed by the imaging element is represented by a large number ofimages corresponding to the respective microlenses included in thearray. Hence, before the movement vector calculation processing and thealignment processing described above are performed, one whole imageneeds to be generated by combining the large number of imagesphotographed by the imaging element.

However, the large number of images corresponding to the respectivemicrolenses are all images inverted vertically and horizontally, and inmany cases, overlap image regions photographed in a duplicate manner arepresent in one microlens photographing image and adjacent microlensphotographing images.

Hence, the generation of one whole image needs a large number of piecesof processing that include, for example, performing vertical andhorizontal inversion processing of each of a large number of microlensphotographing images corresponding to the respective microlensesincluded in the array, performing an image cutout that detects overlapregions of each image and deletes the overlap images, and combining theimages after the cutout with one another.

In order to generate a synthetic image of a high quality, these piecesof processing need to be performed in each frame unit of the pluralityof consecutive photographing images, so that a processing time and aprocessing load become enormous.

CITATION LIST Patent Literature [PTL 1]

-   Japanese Patent Laid-Open No. 2010-240215

SUMMARY Technical Problems

The present disclosure has been made in view of the above-describedproblems, for example. It is an object of the present disclosure toprovide an image processing device, an image processing method, and aprogram that make it possible to perform efficient movement vector(optical flow) detection from photographing images for which a microlensarray (MLA) imaging device is used.

In addition, in one embodiment of the present disclosure, it is anobject to provide an image processing device, an image processingmethod, and a program that make it possible to perform efficientmovement vector (optical flow) detection from photographing images forwhich a microlens array (MLA) imaging device is used, and generate ahigh quality image.

Solution to Problems

According to a first aspect of the present disclosure, there is providedan image processing device including:

a movement vector calculating section configured to receive, as aninput, consecutive photographing images of an imaging section having amicrolens array and calculate a movement vector between the images,

the movement vector calculating section calculating the movement vectorcorresponding to a feature point by using microlens photographing imagesphotographed by respective microlenses included in the microlens array.

Further, according to a second aspect of the present disclosure, thereis provided an image processing device for performing fingerprintauthentication processing, the image processing device including:

a movement vector calculating section configured to receive, as aninput, consecutive photographing images of a finger photographed by animaging section having a microlens array and calculate a movement vectorbetween the images;

an image synthesizing section configured to generate a high qualityimage by synthesis processing of the consecutive photographing images byusing the movement vector calculated by the movement vector calculatingsection; and

an image verifying section configured to perform processing of verifyinga synthetic image generated by the image synthesizing section against aregistered fingerprint image stored in a storage section in advance.

Further, according to a third aspect of the present disclosure, there isprovided an image processing method performed in an image processingdevice, the image processing method including:

by a movement vector calculating section, a movement vector calculatingstep of receiving, as an input, consecutive photographing images of animaging section having a microlens array and calculating a movementvector between the images, in which

the movement vector calculating step includes a step of calculating themovement vector corresponding to a feature point by using microlensphotographing images photographed by respective microlenses included inthe microlens array.

Further, according to a fourth aspect of the present disclosure, thereis provided an image processing method for performing fingerprintauthentication processing in an image processing device, the imageprocessing method including:

by a movement vector calculating section, a movement vector calculatingstep of receiving, as an input, consecutive photographing images of afinger photographed by an imaging section having a microlens array andcalculating a movement vector between the images;

by an image synthesizing section, an image synthesizing step ofgenerating a high quality image by synthesis processing of theconsecutive photographing images by using the movement vector calculatedby the movement vector calculating section; and

by an image verifying section, an image verifying step of performingprocessing of verifying a synthetic image generated by the imagesynthesizing section against a registered fingerprint image stored in astorage section in advance.

Further, according to a fifth aspect of the present disclosure, there isprovided a program for making image processing performed in an imageprocessing device, the program including:

making a movement vector calculating section perform a movement vectorcalculating step of receiving, as an input, consecutive photographingimages of an imaging section having a microlens array and calculating amovement vector between the images, in which

in the movement vector calculating step, processing of calculating themovement vector corresponding to a feature point by using microlensphotographing images photographed by respective microlenses included inthe microlens array is made to be performed.

Further, according to a sixth aspect of the present disclosure, there isprovided a program for making fingerprint authentication processingperformed in an image processing device, the program including:

making a movement vector calculating section perform a movement vectorcalculating step of receiving, as an input, consecutive photographingimages of a finger photographed by an imaging section having a microlensarray and calculating a movement vector between the images;

making an image synthesizing section perform an image synthesizing stepof generating a high quality image by synthesis processing of theconsecutive photographing images by using the movement vector calculatedby the movement vector calculating section; and

making an image verifying section perform an image verifying step ofperforming processing of verifying a synthetic image generated by theimage synthesizing section against a registered fingerprint image storedin a storage section in advance.

Incidentally, the programs according to the present disclosure are, forexample, programs that can be provided by a storage medium or acommunication medium provided in a computer readable form to aninformation processing device or a computer system capable of executingvarious program codes. Processing corresponding to the programs isimplemented on the information processing device or the computer systemby providing such programs in a computer readable form.

Still other objects, features, and advantages of the present disclosurewill become apparent from more detailed description based on embodimentsof the present disclosure to be described later and the accompanyingdrawings. Incidentally, a system in the present specification is alogical set configuration of a plurality of devices and is not limitedto a system in which the devices of respective configurations arelocated within the same casing.

Advantageous Effects of Invention

According to a configuration based on one embodiment of the presentdisclosure, a device and a method are implemented which perform movementvector calculation processing from photographing images for which amicrolens array is used without generating whole images.

Specifically, for example, consecutive photographing images photographedby an imaging section having a microlens array are input, and a movementvector between the images is calculated. A movement vector calculatingsection calculates the movement vector corresponding to a feature pointby using microlens photographing images photographed by respectivemicrolenses. When the movement vector corresponding to the feature pointwithin duplicate image regions having the same image region within aplurality of microlens photographing images is calculated, an averagevalue of a plurality of movement vectors corresponding to the samefeature point is calculated and is set as the movement vector of thefeature point. Alternatively, an average value of plural movementvectors remaining after an outlier (abnormal value) is excluded iscalculated and is set as the movement vector of the feature point.

A device and a method that perform movement vector calculationprocessing from photographing images for which a microlens array is usedwithout generating whole images are implemented by these pieces ofprocessing.

It is to be noted that effects described in the present specificationare merely illustrative and are not limited, and that there may beadditional effects.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram of assistance in explaining an example of aconfiguration of an imaging device having a microlens array (MLA).

FIG. 2 is a diagram of assistance in explaining an example offingerprint authentication processing using the imaging device havingthe microlens array (MLA).

FIG. 3 is a diagram of assistance in explaining a photographing image ofthe imaging device having the microlens array (MLA).

FIG. 4 is a diagram of assistance in explaining a photographing image ofthe imaging device having the microlens array (MLA).

FIG. 5 is a diagram of assistance in explaining movement vectorcalculation processing.

FIG. 6 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of a microlensarray (MLA) imaging device which processing is performed by an imageprocessing device according to the present disclosure.

FIG. 7 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of the microlensarray (MLA) imaging device which processing is performed by the imageprocessing device according to the present disclosure.

FIG. 8 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of the microlensarray (MLA) imaging device which processing is performed by the imageprocessing device according to the present disclosure.

FIG. 9 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of the microlensarray (MLA) imaging device which processing is performed by the imageprocessing device according to the present disclosure.

FIG. 10 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of the microlensarray (MLA) imaging device which processing is performed by the imageprocessing device according to the present disclosure.

FIG. 11 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of the microlensarray (MLA) imaging device which processing is performed by the imageprocessing device according to the present disclosure.

FIG. 12 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of the microlensarray (MLA) imaging device which processing is performed by the imageprocessing device according to the present disclosure.

FIG. 13 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of the microlensarray (MLA) imaging device which processing is performed by the imageprocessing device according to the present disclosure.

FIG. 14 is a diagram of assistance in explaining movement vectorcalculation processing based on photographing images of the microlensarray (MLA) imaging device which processing is performed by the imageprocessing device according to the present disclosure.

FIG. 15 is a diagram of assistance in explaining a fingerprintauthentication processing sequence performed by the image processingdevice according to the present disclosure.

FIG. 16 is a diagram of assistance in explaining the fingerprintauthentication processing sequence performed by the image processingdevice according to the present disclosure.

FIG. 17 is a diagram of assistance in explaining the fingerprintauthentication processing sequence performed by the image processingdevice according to the present disclosure.

FIG. 18 is a diagram of assistance in explaining an example of aconfiguration of an image processing device according to the presentdisclosure.

FIG. 19 is a diagram illustrating a flowchart of assistance inexplaining a sequence of processing performed by the image processingdevice according to the present disclosure.

FIG. 20 is a diagram of assistance in explaining an example of aconfiguration of an image processing device according to the presentdisclosure.

FIG. 21 is a diagram illustrating a flowchart of assistance inexplaining a sequence of processing performed by the image processingdevice according to the present disclosure.

FIG. 22 is a diagram of assistance in explaining an example of aconfiguration of an image processing device according to the presentdisclosure.

FIG. 23 is a diagram illustrating a flowchart of assistance inexplaining a sequence of processing performed by the image processingdevice according to the present disclosure.

FIG. 24 is a diagram of assistance in explaining concrete examples ofdevices that perform the movement vector calculation processingaccording to the present disclosure.

FIG. 25 is a diagram of assistance in explaining concrete examples ofdevices that perform the movement vector calculation processingaccording to the present disclosure.

FIG. 26 is a diagram of assistance in explaining concrete examples ofdevices that perform the movement vector calculation processingaccording to the present disclosure.

FIG. 27 is a diagram of assistance in explaining an example of hardwareconfiguration of the image processing device.

DESCRIPTION OF EMBODIMENTS

In the following, referring to the drawings, description will be made ofdetails of an image processing device, an image processing method, and aprogram according to the present disclosure.

Incidentally, the description will be made according to the followingitems.

1. Microlens Array Imaging Device

2. Fingerprint Authentication Processing Using Microlens Array ImagingDevice and Problems Thereof

3. Movement Vector Calculation Processing in Image Processing Deviceaccording to Present Disclosure

4. Fingerprint Authentication Processing to which Image ProcessingDevice according to Present Disclosure is Applied

5. Examples of Configuration and Processing Sequences of ImageProcessing Devices according to Present Disclosure

6. Concrete Device Examples Using Movement Vector Calculation Processingaccording to Present Disclosure

7. Example of Hardware Configuration of Image Processing Device

8. Summary of Configuration according to Present Disclosure

[1. Microlens Array Imaging Device]

A microlens array imaging device will first be described with referenceto FIG. 1 and following figures.

A microlens array (MLA: Micro Lens Array) imaging device is an imagingdevice in which a plurality of microlenses is arranged.

FIG. 1 illustrates (a) a plan view (top view) and (b) a sectional viewof a microlens array imaging device. As illustrated in (a) the plan view(top view), a large number of small lenses are mounted on the uppersurface of the microlens array imaging device. That is, a microlensarray (MLA: Micro Lens Array) 10 in which microlenses are arranged isformed.

As illustrated in the (b) sectional view, light incident via microlenses11 included in the microlens array (MLA) 10 is applied to and imaged byan image sensor (imaging element) 12.

In the image sensor (imaging element) 12, photographing images of therespective microlenses 11 included in the microlens array (MLA) 10 aregenerated individually.

[2. Fingerprint Authentication Processing Using Microlens Array ImagingDevice and Problems Thereof]

Fingerprint authentication processing using the microlens array imagingdevice and problems thereof will next be described with reference toFIG. 2 and following figures.

FIG. 2 is a diagram of assistance in explaining a sequence offingerprint authentication processing using the microlens array (MLA)imaging device.

The sequence of the fingerprint authentication processing is, forexample, the processing of steps S11 to S14 illustrated in FIG. 2.

First, a moving image of a finger of a user is photographed in step S11.

FIG. 2 illustrates a microlens array (MLA) photographing image @t1, 21photographed at time (t1) and a microlens array (MLA) photographingimage @t2, 22 photographed at time (t2). Incidentally, while an exampleof processing using two consecutive photographing images will bedescribed in the following, three or more consecutive photographingimages may also be photographed and used.

Each of quadrangles illustrated within the microlens array (MLA)photographing images 21 and 22 illustrated in step S11 in FIG. 2 is onemicrolens photographing image 23.

As described earlier, in the microlens array (MLA) imaging device, animage photographed by the imaging element is represented by a largenumber of images corresponding to the respective microlenses included inthe array.

In next step S12, synthesis processing of each of the microlens array(MLA) photographing images 21 and 22 is performed.

The individual microlens photographing images 23 included in themicrolens array (MLA) photographing images 21 and 22 are all imagesinverted vertically and horizontally. Further, in many cases, onemicrolens photographing image and adjacent microlens photographingimages have overlap image regions photographed in a duplicate manner.

Hence, in order to generate one whole image by the synthesis processingof step S12, a large number of pieces of processing are necessary, whichinclude performing vertical and horizontal inversion processing of eachof the microlens photographing images corresponding to the respectivemicrolenses included in the array, performing an image cutout thatdetects overlap regions of each image and deletes the overlap images,and combining the images after the cutout with one another, for example.

This processing needs to be performed for each of the microlens array(MLA) photographing images 21 and 22 as consecutive photographingimages.

As a result of this synthesis processing, two whole images, that is, asynthetic whole image @t1, 31 and a synthetic whole image @t2, 32, aregenerated. These images are images of the finger consecutivelyphotographed at time t1 and time t2.

By using the two images, that is, the synthetic whole image @t1, 31 andthe synthetic whole image @t2, 32, step S13 performs processing ofdetecting a movement vector (optical flow) of a feature point within theimages.

This is processing of finding the feature point estimated to be a commonsubject from the synthetic whole image @t1, 31 and the synthetic wholeimage @t2, 32, and calculating an amount of movement and a movingdirection between the two images.

Next, step S14 performs alignment between the two images, that is, thesynthetic whole image @t1, 31 and the synthetic whole image @t2, 32, onthe basis of the movement vector calculated in step S13, and generates ahigh quality image by synthesis processing of the two images. Forexample, alpha blending processing of pixel values of respectivecorresponding pixels or the like is performed to generate a clearer highquality image.

A high quality image 35 having clear fingerprint information of thefinger or the like is generated as a result of this processing.

Though not illustrated in the figure, processing of verifying the highquality image 35 against fingerprint data registered in a memory inadvance is finally performed.

The synthesis of the plural images is performed because, for example,only one still image has insufficient characteristic information of thefinger and thus decreases authentication accuracy. The synthesis of aplurality of consecutive photographing images needs the alignment of theplurality of consecutive photographing image frames. This alignmentneeds the processing of calculating the movement vector (optical flow)of a feature point included in the images.

Processing using the two consecutively photographed whole images isperformed in the calculation of the movement vector (optical flow) instep S13. The synthesis processing of step S12 is performed for thiswhole image generation processing.

As described above, the generation of one whole image needs vertical andhorizontal inversion processing performed for each of the microlensphotographing images corresponding to the respective microlensesincluded in the microlens array (MLA).

Further, in many cases, overlap image regions are present in each of themicrolens photographing images included in the microlens array (MLA).Hence, high-load processing is necessary, which includes performing animage cutout that detects overlap regions of each image and deletes theoverlap images and combining the images after the cutout with oneanother.

Referring to FIG. 3, description will be made of overlap image regionsoccurring in the images corresponding to the respective microlensesincluded in the microlens array (MLA).

FIG. 3(1) illustrates (b) a sectional view of a microlens array imagingdevice similar to that described with reference to FIG. 1(b). A part oflight incident on a microlens included in the microlens array (MLA)includes a region overlapping light incident on an adjacent microlens.As a result, overlap image regions photographed in a duplicate manneroccur in one microlens photographing image and an adjacent microlensphotographing image.

FIG. 3(2) illustrates a microlens array 40, a photographing region 41,and a photographing image 42. The photographing region 41 includesalphabetic characters, numerical values, and symbols. The photographingimage 42 is a result of photographing the photographing region 41. Thephotographing image 42 represents 16 microlens photographing images of4×4 microlenses in an upper left portion of the microlens array 40.

As is understood from the photographing image 42 in FIG. 3(2), each ofthe microlens photographing images included in the photographing image42 is photographed as an image inverted vertically and horizontally.Further, overlap image regions photographed in a duplicate manner occurin one microlens photographing image and adjacent microlensphotographing images.

For example, a substantially whole image of an alphabetic characterimage “A, B” and a part of a numerical value image “2, 3” arephotographed in a microlens photographing image at an upper left end ofthe photographing image 42. An image of half of an alphabetic characterimage “B” is photographed in an adjacent microlens photographing imageon the right of the microlens photographing image at the upper left end.That is, at least a part of the alphabetic character image “B” is anoverlap image photographed in both images of the two microlensphotographing images.

In addition, approximately an upper half of the numerical value image“2, 3” is photographed in the microlens photographing image at the upperleft end of the photographing image 42, while more than half of a lowerportion of the numerical value image “2, 3” is photographed in anadjacent microlens photographing image below the microlens photographingimage at the upper left end. That is, at least a part of the image “2,3” is an overlap image photographed in both images of the two microlensphotographing images.

Manners of occurrence of overlap images in microlens photographingimages will be described with reference to FIG. 4.

Overlap image regions occurring in one microlens photographing image 45illustrated in FIG. 4 will be described.

The one microlens photographing image 45 illustrated in FIG. 4 hasoverlap image regions such that each of images of upper and lower andleft and right side regions is also included in another microlensphotographing image.

An upper edge region image (U-image) of the microlens photographingimage 45 is an overlap image identical to a lower edge region image ofanother microlens photographing image adjacent to the microlensphotographing image 45 illustrated in the figure in a downwarddirection.

A lower edge region image (D-image) of the microlens photographing image45 is an overlap image identical to an upper edge region image ofanother microlens photographing image adjacent to the microlensphotographing image 45 illustrated in the figure in an upward direction.

A left edge region image (L-image) of the microlens photographing image45 is an overlap image identical to a right edge region image of anothermicrolens photographing image adjacent to the microlens photographingimage 45 illustrated in the figure in a right direction.

A right edge region image (R-image) of the microlens photographing image45 is an overlap image identical to a left edge region image of anothermicrolens photographing image adjacent to the microlens photographingimage 45 illustrated in the figure in a left direction.

Overlap image regions are thus present in each of the microlensphotographing images. Hence, the generation of a whole image needsprocessing such as performing an image cutout that detects overlapregions of each image and deletes the overlap images and combining theimages after the cutout with one another. These pieces of processingimpose a high load and cause problems such as the occurrence of aprocessing delay and an increase in processing cost.

[3. Movement Vector Calculation Processing in Image Processing DeviceAccording to Present Disclosure]

Movement vector calculation processing in an image processing deviceaccording to the present disclosure will next be described.

As described earlier with reference to FIG. 2, movement vector (opticalflow) detection processing is performed to detect an identical subjectposition (corresponding pixel position) by detecting a manner ofmovement of a corresponding point (feature point) between two imagesphotographed in two different timings, for example. After thecorresponding pixel positions of plural images are calculated, one highquality image is generated by synthesis processing of the plural images.

An example of ordinary movement vector calculation processing will bedescribed with reference to FIG. 5.

FIG. 5 is a diagram illustrating an example of movement vector (opticalflow) calculation processing using a photographing image of an ordinarysingle lens camera rather than an imaging device using microlenses.

FIG. 5 illustrates two consecutive photographing image frames f1 and f2.

The image frame f1 is a photographing image at time t1. The image framef2 is a photographing image at immediately succeeding time t2.

A feature point a, 51 and a feature point b, 52 determined that they areobtained by photographing an identical subject are photographed in theframe f1 and the frame f2.

As illustrated in the frame f2, a movement vector is a vector having afeature point position in the frame f1 as a starting point and havingthe identical feature point position in the frame f2 as an end point.That is, the movement vector (optical flow) is a vector indicating amoving direction and an amount of movement of a feature point from thetime of photographing in the frame f1 to the time of photographing inthe frame f2.

In the case where photographing images of an ordinary camera are used,whole images at respective photographing times can be obtainedconsecutively, and the calculation of the movement vector can be thusperformed relatively easily.

However, there is a problem that an imaging device using microlensesincreases the processing load of processing that generates whole imagesfor use in movement vector calculation as described earlier withreference to FIGS. 2 to 4 and consequently increases the processing loadof the movement vector calculation processing between the consecutivephotographing images.

Movement vector calculation processing according to the presentdisclosure for solving this problem will be described in the following.

In processing according to the present disclosure, the movement vectorcalculation processing is performed by using a photographing imageitself of the imaging device using the microlenses, that is, a largenumber of microlens photographing images, without generating one wholeimage from the photographing image of the imaging device using themicrolenses.

A concrete example of processing performed by the image processingdevice according to the present disclosure will be described withreference to FIG. 6 and following figures.

FIG. 6 illustrates a part of the following two consecutive photographingimage frames of the microlens array (MLA) imaging device.

(1) a microlens array (MLA) photographing image @t1 (frame f1)

(2) a microlens array (MLA) photographing image @t2 (frame f2)

(1) the microlens array (MLA) photographing image @t1 (frame f1) is animage photographed at time t1, and (2) the microlens array (MLA)photographing image @t2 (frame f2) is an image photographed atimmediately succeeding time t2.

Incidentally, the figure illustrates a part of each frame image, thatis, 3×3=9 microlens photographing images. (1) and (2) are both imagesphotographed by nine microlenses at identical positions.

At the time of calculation of a movement vector, a feature point isfirst extracted from the image in the frame f1. The example illustratedin FIG. 6 represents an example in which a feature point a, 102 isextracted from a central region image (C-image) of a central microlensphotographing image 101 in the frame f1.

Incidentally, the central region image (C-image) of the microlensphotographing image 101 is not an overlap image region. That is, thiscentral region image (C-image) is a single image region not included inother peripheral microlens photographing images.

In this case, a feature point search is performed after a search rangein the frame 2 for the feature point a, 102 detected from the frame f1is set to the central region image (C-image) of the same microlensphotographing image 101 as the microlens photographing image 101 inwhich the feature point a, 102 is detected.

That is, the feature point search is performed after the search range isset in which the position of the lens and the manner of inversion of theimage are taken into consideration.

Suppose that this feature point search detects the feature point a, 102in the central region image (C-image) of the microlens photographingimage 101 in the frame f2 of FIG. 6(2).

Consequently, a movement vector 103 illustrated in the central region(C-image) of the microlens photographing image 101 in the frame f2 ofFIG. 6(2) is calculated.

The image processing device according to the present disclosure sets acorresponding feature point detection range (search range) in the frame2 according to the position of the feature point detected from the framef1. In the example illustrated in FIG. 6, the feature point a, 102detected from the frame f1 is in the central region (C-image) of themicrolens photographing image 101 and is not in an overlap image region.Thus, the search range in the frame 2 is limited to the central regionimage (C-image) of the same microlens photographing image 101 as themicrolens photographing image 101 in which the feature point a, 102 isdetected.

The limitation of the search range makes high-speed processing possible.

Next, referring to FIG. 7, description will be made of processing in acase where a feature point is present in an overlap image region(duplicate image region).

As with FIG. 6, FIG. 7 also illustrates a part of the following twoconsecutive photographing image frames of the microlens array (MLA)imaging device.

(1) a microlens array (MLA) photographing image @t1 (frame f1)

(2) a microlens array (MLA) photographing image @t2 (frame f2)

(1) the microlens array (MLA) photographing image @t1 (frame f1) is animage photographed at time t1, and

(2) the microlens array (MLA) photographing image @t2 (frame f2) is animage photographed at immediately succeeding time t2. (1) and (2) areboth images photographed by nine microlenses at identical positions.

At the time of calculation of a movement vector, a feature point isfirst extracted from the image in the frame f1. The example illustratedin FIG. 7 represents an example in which a feature point a, 121 isextracted from a lower region image (D-image) of a central microlensphotographing image 101 in the frame f1.

Here, the lower region image (D-image) of the microlens photographingimage 101 is an overlap image region. That is, this lower region image(D-image) is an overlap image region included also in an upper adjacentmicrolens photographing image 111 adjacent to the microlensphotographing image 101 illustrated in the figure in an upwarddirection.

In this case, the same feature point a, 121 as the feature point a, 121detected from the frame f1 can be detected from an upper region image(U-image) of the upper adjacent microlens photographing image 111.

In this case, a feature point search is performed after a search rangein the frame 2 is set to not only the lower region image (D-image) ofthe same microlens photographing image 101 as the microlensphotographing image 101 in which the feature point a, 121 is detectedbut also the upper region image (U-image) of the same upper adjacentmicrolens photographing image 111 as the upper adjacent microlensphotographing image 111 as an overlap image region in which the samefeature point a, 121 is detected.

That is, the feature point search is performed after the search range isset in which the positions of the lenses, the manner of inversion of theimages, and the image duplicate regions are taken into consideration.

Suppose that this feature point search detects the feature point a, 121from the lower region image (D-image) of the microlens photographingimage 101 and the upper region image (U-image) of the upper adjacentmicrolens photographing image 111 in the frame f2 of FIG. 7(2).

Consequently, a movement vector 113(1) corresponding to the featurepoint a, 121 is set in the lower region image (D-image) of the microlensphotographing image 101 in the frame f2 of FIG. 7(2), and a movementvector 113(2) corresponding to the same feature point a, 121 is also setin the upper region image (U-image) of the upper adjacent microlensphotographing image 111.

These movement vectors correspond to one identical feature point a, 121and therefore need to be ultimately set to be one movement vector.Processing of calculation of this one final movement vector will bedescribed in a later paragraph.

The above description has been made of an example in which, in a casewhere the same image is included in different microlens photographingimages and the same feature point is detected therein, two movementvectors corresponding to the one identical feature point can be detectedfrom these two image regions.

In a case where the position of a feature point detected from the framef1 is in an overlap image region, the image processing device accordingto the present disclosure sets the corresponding feature point detectionrange (search range) in the frame 2 to a plurality of overlap imageregions identical to the detection region of the feature point detectedfrom the frame f1. The limitation of the search range makes high-speedprocessing possible.

Incidentally, the example illustrated in FIG. 7 is an example in whichone identical feature point a, 121 is detected from two differentmicrolens photographing images. However, there are cases where oneidentical feature point is detected from four different microlensphotographing images, depending on the detected position of the featurepoint.

That is, there are cases where the same image region is included in fourdifferent microlens photographing images.

This example will be described with reference to FIG. 8.

Nine microlens photographing images are also illustrated in FIG. 8.

Each of images of image regions at four vertices of a central microlensphotographing image 101 is included in four different microlensphotographing images.

Description will be made of an upper left end region image 101UL of themicrolens photographing image 101 as one of the four vertex regionimages in FIG. 8.

As for the upper left end region image 101UL of the microlensphotographing image 101, images identical to this region image are alsoincluded in three other microlens photographing images different fromthe microlens photographing image 101.

The identical images are included in regions indicated by circle marksin the figure, that is, the following three regions.

(1) an upper right end region image 121UR of a microlens photographingimage 121 adjacent on the right of the microlens photographing image 101

(2) a right lower end region image 122DR of a microlens photographingimage 122 adjacent on the lower right of the microlens photographingimage 101

(3) a lower left end region image 123DL of a microlens photographingimage 123 adjacent below the microlens photographing image 101

Thus, the upper left end region image 101UL of the microlensphotographing image 101 is also present in the three other differentmicrolens photographing images 121, 122, and 123. That is, the sameimage region is included in the four different microlens photographingimages.

Incidentally, while description has been made of only one of the fourvertices of the central microlens photographing image 101 in FIG. 8, thefour vertex images are all similar and are image regions included infour different microlens photographing images.

In a case where the same image is thus included in four image regionsand the image processing device according to the present disclosuredetects a feature point in the regions, the image processing deviceaccording to the present disclosure performs processing of detectingfour movement vectors corresponding to the same feature point, in all ofthe four image regions including the same image.

Incidentally, the search range of the feature point is limited to thefour photographing image regions of four identical microlenses in eachframe.

That is, a feature point search is performed after a search range is setin which the positions of the lenses, the manner of inversion of theimages, and the image duplicate regions are taken into consideration.

Next, referring to FIG. 9 and following figures, description will bemade of a concrete example of movement vector calculation processing.

The following three types of movement vector calculation processing willbe described below in order.

(1) movement vector calculation processing corresponding to a featurepoint in a single image region

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions

First, (1) movement vector calculation processing corresponding to afeature point in a single image region will be described with referenceto FIG. 9(1).

This movement vector calculation processing corresponds to theprocessing described earlier with reference to FIG. 6.

That is, this movement vector calculation processing is the processingof calculating a movement vector corresponding to a feature pointdetected from the central region image (C-image) of the microlensphotographing image that is not an overlap image region. In the presentexample, only one movement vector corresponding to the same featurepoint can be detected.

The central region image (C-image) of the microlens photographing imageis not an overlap image region. The central region image (C-image) is asingle image region not included in other peripheral microlensphotographing images.

In this case, a feature point search is performed after the centralregion image (C-image) of the same microlens photographing image in theframe f2 as the microlens photographing image in the frame f1 in whichthe feature point is detected is set as a search range. In a case wherethe feature point search detects the feature point from the frame f2, avector is calculated which has a feature point position (x1, y1) in theframe f1 as a starting point and has a feature point position (x2, y2)in the frame f2 as an end point.

Consequently, as illustrated in FIG. 9(1), movement vector V=(Vx,Vy)=(x2−x1, y2−y1) can be calculated.

Next, (2) movement vector calculation processing corresponding to afeature point in two-image duplicate regions will be described withreference to FIG. 9(2).

This movement vector calculation processing corresponds to theprocessing described earlier with reference to FIG. 7.

That is, this movement vector calculation processing is the processingof calculating a movement vector in edge regions (excluding four vertexvicinity regions) of microlens photographing images as overlap imageregions in which two identical images are present. In the presentexample, two movement vectors corresponding to the same feature pointcan be detected.

The edge regions (excluding four vertex vicinity regions) of themicrolens photographing images are overlap image regions and aretwo-image duplicate regions such that the same image is included inanother microlens photographing image.

In this case, a feature point search is performed after the same imageregions (overlap image regions) of two microlens photographing images inthe frame f2 as the image regions (overlap image regions) of twomicrolens photographing images in the frame f1 in which the same featurepoint is detected are set as a search range. This feature point searchmay detect two movement vectors corresponding to two identical featurepoints from the frame f2.

With regard to the two movement vectors, one final movement vector iscalculated by processing illustrated in FIG. 9(2) (step S51).

That is, an average value V=(Vx, Vy) of the two movement vectorscorresponding to the same feature point is calculated according to(Equation 1) in the following.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\{\overset{\_}{v_{x}} = {{\frac{1}{2}{\sum\limits_{i = 1}^{2}{v_{x_{i}}\mspace{31mu}\overset{\_}{v_{y}}}}} = {\frac{1}{2}{\sum\limits_{i = 1}^{2}v_{y_{i}}}}}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

The average value V=(Vx, Vy) of the two movement vectors correspondingto the same feature point, the average value being calculated accordingto the above-described (Equation 1), is calculated as a movement vector.

Next, (3) movement vector calculation processing corresponding to afeature point in four-image duplicate regions will be described withreference to FIG. 10(3).

This movement vector calculation processing corresponds to processing ina case where the same image is included in four microlens photographingimages, as described earlier with reference to FIG. 8.

That is, this movement vector calculation processing is the processingof calculating a movement vector in four vertex vicinity regions ofmicrolens photographing images as overlap image regions in which fouridentical images are present. In the present example, four movementvectors corresponding to the same feature point can be detected.

The four vertex vicinity regions of the microlens photographing imagesare overlap image regions and are four-image duplicate regions such thatthe same image is included in the four microlens photographing images.

In this case, a feature point search is performed after the same imageregions (overlap image regions) of four microlens photographing imagesin the frame f2 as the image regions (overlap image regions) of fourmicrolens photographing images in the frame f1 in which the same featurepoint is detected are set as a search range. This feature point searchmay detect four movement vectors corresponding to four identical featurepoints from the frame f2.

With regard to the four movement vectors, one final movement vector iscalculated by processing illustrated in FIG. 10(3) (steps S71 to S73).

First, in step S71, an average value V=(Vx, Vy) of the four movementvectors corresponding to the same feature point is calculated accordingto (Equation 2) in the following.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{\overset{\_}{v_{x}} = {{\frac{1}{4}{\sum\limits_{i = 1}^{4}{v_{x_{i}}\mspace{31mu}\overset{\_}{v_{y}}}}} = {\frac{1}{4}{\sum\limits_{i = 1}^{4}v_{y_{i}}}}}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$

Next, in step S72, processing of excluding an outlier (abnormal value)from the four movement vectors is performed.

This processing is performed according to (Equation 3) in the following.

[Math. 3]

i satisfying (√{square root over (δ_(xi) ²+δ_(yi) ²)}>δ_(th)) at is setas an outlier

where

(δ_(xi) =v _(xi)− v _(x) ,δ_(yi) =v _(yi)− v _(y) (i=1, . . .4)  (Equation 3)

Next, in step S73, the outlier (abnormal value) detected in step S72 isexcluded, and an average value V=(Vx, Vy) of the remaining movementvectors is calculated again.

Consequently, the calculated average value V=(Vx, Vy) is set as a finalmovement vector corresponding to the feature point.

Thus, the image processing device according to the present disclosureperforms the three types of movement vector calculation processingdescribed with reference to FIGS. 9 to 10, that is, the following threekinds of processing.

(1) movement vector calculation processing corresponding to a featurepoint in a single image region

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions

In the following pieces of processing:

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions, and

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions,

among the above-described pieces of processing, one final movementvector corresponding to a feature point is calculated by using aplurality of movement vectors corresponding to the same feature point,and thus, more accurate movement vector calculation can be performed byperforming the processing not dependent on one data but based on pluralpieces of data.

Incidentally, which of the above-described pieces of processing (1) to(3) to perform is determined according to the position of the featurepoint detected by a data processing section of the image processingdevice.

In a case where the feature point is detected from the central regionimage (C-image) of a microlens photographing image,

(1) movement vector calculation processing corresponding to a featurepoint in a single image region is performed.

In addition, in a case where the feature point is detected from edgeregions (excluding four vertex vicinity regions) of microlensphotographing images as overlap image regions in which two identicalimages are present,

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions is performed.

Further, in a case where the feature point is detected from four vertexvicinity regions of microlens photographing images as overlap imageregions in which four identical images are present,

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions is performed.

Exceptionally, however, different processing is performed as processingfor feature points detected from microlens photographing imagescorresponding to microlenses present at outermost circumferentialpositions of the microlens array.

This exception processing will be described with reference to FIG. 11and following figures.

The left side of FIG. 11 illustrates the microlens array (MLA) 10.Microlenses present at outermost circumferential positions of themicrolens array (MLA) 10 are illustrated in white.

Consideration will be given to a photographing image of one microlensa150 among the microlenses, for example.

The right side of FIG. 11 illustrates a microlens “a” photographingimage 151 as a photographing image of the microlens a150 present at anoutermost circumferential position of the microlens array (MLA) 10.

An upper region image (U-image) of the microlens “a” photographing image151 is an overlap image identical to a lower edge region image ofanother microlens photographing image adjacent to the microlens “a”photographing image 151 illustrated in the figure in the downwarddirection.

In addition, a lower edge region image (D-image) of the microlens “a”photographing image 151 is an overlap image identical to an upper edgeregion image of another microlens photographing image adjacent to themicrolens “a” photographing image 151 illustrated in the figure in theupward direction.

A left edge region image (L-image) of the microlens “a” photographingimage 151 is an overlap image identical to a right edge region image ofanother microlens photographing image adjacent to the microlens “a”photographing image 151 illustrated in the figure in the rightdirection.

However, because no microlens photographing image is present on the leftside of the microlens “a” photographing image 151, a right edge regionimage (R-image) of the microlens “a” photographing image 151 is a singleimage whose identical image is not present in other microlensphotographing images.

In this case, in a case where the image processing device according tothe present disclosure detects a feature point from the right edgeregion image (R-image) of the microlens “a” photographing image 151, asin a case where the image processing device detects a feature point fromthe central region image (C-image) of a microlens photographing image,the image processing device performs the processing illustrated in FIG.9(1), that is,

(1) movement vector calculation processing corresponding to a featurepoint in a single image region.

FIG. 12 is a diagram illustrating the microlens “a” photographing image151 as a photographing image of the microlens a150 present at anoutermost circumferential position of the microlens array (MLA) 10 as inFIG. 11.

A diagram on the right side of FIG. 12 is a diagram of assistance inexplaining an overlap mode of an upper right region image (UR-image)151UR of the microlens “a” photographing image 151. In a case where themicrolens photographing image is not at an outermost circumferentialposition of the microlens array (MLA) 10, four duplicate images arepresent as four vertex vicinity images as described earlier withreference to FIG. 8, and (3) movement vector calculation processingcorresponding to a feature point in four-image duplicate regions isperformed, which has been described with reference to FIG. 10(3).

However, as illustrated in FIG. 12, because no microlens photographingimage is present on the left side of the microlens “a” photographingimage 151, an overlap image of the upper right region image (UR-image)151UR of the microlens “a” photographing image 151 is present only in amicrolens photographing image 152 adjacent on the lower side of themicrolens “a” photographing image 151.

That is, there are only two duplicate images.

Hence, in a case where the image processing device according to thepresent disclosure detects a feature point from a vertex vicinity regionimage of the microlens “a” photographing image 151 present at anoutermost circumferential position of the microlens array (MLA) 10, asin a case where the image processing device detects a feature point fromoverlap image regions in which two identical images are present, theimage processing device performs

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions, which is illustrated in FIG. 9(2).

FIG. 13 is a diagram of assistance in explaining processing for aphotographing image of one microlens b160 at a vertex position of themicrolens array (MLA) 10. The right side of FIG. 13 illustrates amicrolens “b” photographing image 161 as a photographing image of themicrolens b160 present at an outermost circumferential position and avertex position of the microlens array (MLA) 10.

An upper region image (U-image) of the microlens “b” photographing image161 is an overlap image identical to a lower edge region image ofanother microlens photographing image adjacent to the microlens “b”photographing image 161 illustrated in the figure in the downwarddirection.

In addition, a left edge region image (L-image) of the microlens “b”photographing image 161 is an overlap image identical to a right edgeregion image of another microlens photographing image adjacent to themicrolens “b” photographing image 161 illustrated in the figure in theright direction.

However, because no microlens photographing image is present on theupper side and the left side of the microlens “b” photographing image161, a lower edge region image (D-image) and a right edge region image(R-image) of the microlens “b” photographing image 161 are each a singleimage whose identical image is not present in other microlensphotographing images.

In this case, in a case where the image processing device according tothe present disclosure detects a feature point from the lower edgeregion image (D-image) or the right edge region image (R-image) of themicrolens “b” photographing image 161, as in a case where the imageprocessing device detects a feature point from the central region image(C-image) of a microlens photographing image, the image processingdevice performs the processing illustrated in FIG. 9(1), that is, (1)movement vector calculation processing corresponding to a feature pointin a single image region.

FIG. 14 is a diagram illustrating the microlens “b” photographing image161 as a photographing image of the microlens b160 present at anoutermost circumferential position and a vertex position of themicrolens array (MLA) 10 as in FIG. 13.

A diagram on the right side of FIG. 14 is a diagram of assistance inexplaining an overlap mode of an upper right region image (UR-image)161UR of the microlens “b” photographing image 161. In a case where themicrolens photographing image is not at an outermost circumferentialposition of the microlens array (MLA) 10, four duplicate images arepresent as four vertex vicinity images as described earlier withreference to FIG. 8, and (3) movement vector calculation processingcorresponding to a feature point in four-image duplicate regions isperformed, which has been described with reference to FIG. 10(3).

However, as illustrated in FIG. 14, because no microlens photographingimage is present on the upper side and the left side of the microlens“b” photographing image 161, an overlap image of the upper right regionimage (UR-image) 161UR of the microlens “b” photographing image 161 ispresent only in a microlens photographing image 162 adjacent on thelower side of the microlens “b” photographing image 161.

That is, there are only two duplicate images.

Hence, in a case where the image processing device according to thepresent disclosure detects a feature point from the upper right regionimage (UR-image) 161UR of the microlens “b” photographing image 161present at an outermost circumferential vertex position of the microlensarray (MLA) 10, as in a case where the image processing device detects afeature point from overlap image regions in which two identical imagesare present, the image processing device performs

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions, which is illustrated in FIG. 9(2).

In addition, because no adjacent microlens photographing image ispresent on the upper side nor the left side of the microlens “b”photographing image 161, an overlap image of a right lower region image(DR-image) 161DR of the microlens “b” photographing image 161 is asingle image region whose duplicate images are not present.

In this case, in a case where the image processing device according tothe present disclosure detects a feature point from the right lowerregion image (DR-image) 161DR of the microlens “b” photographing image161, as in a case where the image processing device detects a featurepoint from the central region image (C-image) of a microlensphotographing image, the image processing device performs the processingillustrated in FIG. 9(1), that is,

(1) movement vector calculation processing corresponding to a featurepoint in a single image region.

Thus, the image processing device according to the present disclosuredetermines how many image regions identical to an image region at aposition at which a feature point is detected are included in microlensphotographing images, selects which of the pieces of processingdescribed with reference to FIG. 9 and FIG. 10, that is,

(1) movement vector calculation processing corresponding to a featurepoint in a single image region,

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions, and

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions

to perform according to the number of identical image regions, andperforms the selected processing.[4. Fingerprint Authentication Processing to which Image ProcessingDevice according to Present Disclosure is Applied]

Next, description will be made of the present fingerprint authenticationprocessing to which the above-described movement vector (optical flow)processing according to the present disclosure is applied.

Incidentally, there are biometric authenticating devices that perform,for example, vein authentication, fingerprint authentication, irisauthentication, and the like as biometric authenticating devices using amicrolens array imaging device. Here, as a representative examplethereof, a fingerprint authenticating device using a microlens arrayimaging device will be described in the following. The configurationaccording to the present disclosure is not limited to the fingerprintauthenticating device but applicable also to other biometricauthenticating devices for vein authentication, iris authentication, andthe like.

FIG. 15 is a diagram of assistance in explaining a sequence of thefingerprint authentication processing by the image processing deviceaccording to the present disclosure. The sequence of the fingerprintauthentication processing is, for example, performed according to theprocessing of steps S101 to S103 illustrated in FIG. 15.

Incidentally, the conventional fingerprint authentication processingsequence using the microlens array imaging device has been describedearlier with reference to FIG. 2. The sequence illustrated in FIG. 15 isa sequence that makes it possible to perform the fingerprintauthentication processing using the same microlens array imaging deviceefficiently without causing a high processing load.

The processing of steps S101 to S103 illustrated in FIG. 15 will bedescribed in order. Incidentally, these pieces of processing can beperformed according to a program stored in a storage section of theimage processing device, under control of a control section (dataprocessing section) including a CPU having a program executing functionor the like.

(Step S101)

First, a moving image of a finger of a user is photographed in stepS101.

FIG. 15 illustrates a microlens array (MLA) photographing image @t1, 201photographed at time (t1) and a microlens array (MLA) photographingimage @t2, 202 photographed at time (t2). Incidentally, while an exampleof processing using two consecutive photographing images will bedescribed in the following, three or more consecutive photographingimages may also be used.

Each of quadrangles illustrated within the microlens array (MLA)photographing images 201 and 202 illustrated in step S101 in FIG. 15 isone microlens photographing image 203.

As described earlier, in the microlens array (MLA) imaging device, animage photographed by the imaging element is represented by a largenumber of images corresponding to the respective microlenses included inthe array.

(Step S102)

Next step S102 performs processing of detecting a movement vector(optical flow) from the two microlens array (MLA) photographing images201 and 202 as two consecutive photographing images.

This movement vector (optical flow) detection processing is the movementvector detection processing according to the present disclosure, whichhas been described earlier with reference to FIGS. 6 to 13. That is, themovement vector (optical flow) detection processing is performed byusing the photographing images themselves including a large number ofmicrolens photographing images, without generating the whole images asin step S12 described earlier with reference to FIG. 2.

The image processing device according to the present disclosure detectsa feature point as a movement vector calculation target from thephotographing images including a large number of microlens photographingimages, and performs the setting of a different search range andmovement vector calculation processing according to the position of thedetected feature point.

Specifically, in a case where a feature point is detected from thecentral region image (C-image) of one microlens photographing image, acorresponding feature point is searched after a central region image(C-image) at the same position in a microlens photographing imagephotographed in different timing is set as a search range. Further,

(1) movement vector calculation processing corresponding to a featurepoint in a single image region is performed, which has been describedearlier with reference to FIG. 9(1).

In addition, in a case where a feature point is detected from edgeregions (excluding four vertex vicinity regions) of microlensphotographing images as overlap image regions in which two identicalimages are present, the same feature point is first detected from theoverlap image regions in the same frame.

Next, a corresponding feature point is searched after two identicalimage regions of microlens photographing images photographed indifferent timing are set as a search range. Further,

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions is performed, which has beendescribed earlier with reference to FIG. 9(2).

Further, in a case where a feature point is detected from four vertexvicinity regions of microlens photographing images as overlap imageregions in which four identical images are present, the same featurepoint is first detected from the overlap image regions in the sameframe.

Next, a corresponding feature point is searched after four identicalimage regions of microlens photographing images photographed indifferent timing are set as a search range. Further,

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions is performed, which has beendescribed earlier with reference to FIG. 10(3).

By performing these pieces of processing, it is possible to detect amovement vector (optical flow) without generating whole images as instep S12 described earlier with reference to FIG. 2.

(Step S103)

Next, by using the movement vector detected in step S102, step S103performs alignment of each image and high quality image generationprocessing. Incidentally, this image alignment processing and the highquality image generation processing can be performed in microlensphotographing image units.

The image alignment processing and the high quality image generationprocessing are performed in microlens photographing image units, andthereafter, one whole image of a high quality is generated by combiningmicrolens photographing images improved in image quality with eachother.

Here, it is necessary to perform the vertical and horizontal inversionprocessing of the individual microlens photographing images, an imagecutout that detects and deletes overlap regions, and image combinationprocessing after the cutout. However, this processing is performed onlyonce in the processing according to the present disclosure.

In the processing described earlier with reference to FIG. 2, theprocessing of generating a whole image for each individual photographingimage is necessary in step S12. On an assumption that the processingusing a minimum of two or more consecutive photographing images isperformed, the number of man-hours of the whole image generationprocessing in the processing according to the present disclosure isequal to or less than ½ that of the conventional processing illustratedin FIG. 2.

A high quality image 210 having clear fingerprint information of thefinger and the like is generated as a result of the processing of stepS103.

Though not illustrated in the figure, processing of verifying the highquality image 210 against fingerprint data registered in a memory inadvance is finally performed.

Incidentally, in the processing according to the present disclosure, ina case where a feature point is detected from duplicate image regions atthe time of the movement vector (optical flow) calculation processing instep S102, one final movement vector corresponding to the feature pointis calculated by using a plurality of movement vectors corresponding tothe same feature point. That is, a configuration that performsprocessing not dependent on one piece of data but based on a pluralityof pieces of data is provided, so that more accurate movement vectorcalculation can be performed.

FIG. 16 and FIG. 17 are diagrams of assistance in explaining a concreteexample of fingerprint authentication processing to which the imageprocessing device according to the present disclosure is applied. FIGS.16 to 17 illustrate concrete examples of the respective pieces ofprocessing of processing steps S101 to S104 of the fingerprintauthentication processing. The processing of steps S101 to S103 isprocessing corresponding to the processing steps S101 to S103 describedwith reference to FIG. 15.

The processing of each step illustrated in FIGS. 16 to 17 will bedescribed.

(Step S101)

Step S101 is a step of photographing a moving image of a finger of auser. As illustrated in FIG. 17, the user slides the finger on thescreen of a smartphone. A microlens array (MLA) imaging device ismounted on the screen of the smartphone.

As illustrated in FIG. 16, three consecutive photographing images inframes f1 to f3 are photographed in conjunction with movement of thefinger.

(Step S102)

Next step S102 is movement vector (optical flow) detection processingusing the three images photographed in step S101.

Step S102 in FIG. 16 illustrates an example of the following movementvector detection results:

a movement vector detection processing result between the frame f1 andthe frame f2, and

a movement vector detection processing result between the frame f2 andthe frame f3.

A synthetic image of a high quality can be generated by thus detectingmovement vectors between two or more consecutive photographing images,setting the frame f2, for example, as a reference image, andsynthesizing the images with the position of the images in the frame f1and the frame f3 adjusted to the position of the reference image (framef2).

The movement vector (optical flow) detection processing illustrated instep S102 in FIG. 16 is performed as processing that uses a large numberof microlens photographing images as they are.

A result of performing the movement vector detection processingaccording to the present disclosure, which has been described withreference to FIGS. 6 to 13, is obtained. According to the position of afeature point detected from a microlens photographing image, any one ofthe three types of movement vector calculation processing describedearlier with reference to FIG. 9 and FIG. 10, that is, the followingthree kinds of processing, is performed.

(1) movement vector calculation processing corresponding to a featurepoint in a single image region

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions

By performing this processing, it is possible to detect a movementvector (optical flow) without generating whole images as in step S12described earlier with reference to FIG. 2.

(Step S103)

The processing of next step S103 is processing of aligning each image byusing the movement vector detected in step S102, and generating a highquality image.

In the example illustrated in FIG. 16, a high quality image is generatedby using the three images in the frames f1 to f3. For example, the imageposition of the frame f1 is adjusted to the position of the frame f2 asa reference image. This alignment processing can be performed by usingthe movement vector between the frame f1 and the frame f2 which movementvector is detected in step S102.

Further, the image position of the frame f3 is adjusted to the positionof the frame f2 as a reference image. This alignment processing can beperformed by using the movement vector between the frame f2 and theframe f3 which movement vector is detected in step S102.

Image quality improvement processing is thereafter performed byprocessing such as alpha blending of corresponding pixels in these threedifferent photographing timings. One whole image of a high quality isthereafter generated by combining the microlens photographing imagesimproved in image quality with one another. Here, it is necessary toperform the vertical and horizontal inversion processing of theindividual microlens photographing images, an image cutout that detectsand deletes overlap regions, and image combination processing after thecutout. However, this processing is only one piece of whole imagegeneration processing in the processing according to the presentdisclosure.

A high quality image having clear fingerprint information of the fingerand the like is generated as a result of the processing of step S103.

(Step S104)

Finally, the fingerprint authentication processing is performed in stepS104 illustrated in FIG. 17. This fingerprint authentication processingis processing of verifying the high quality image generated in step S103against fingerprint data registered in a memory 230 in advance.

The high quality image generated in step S103 is (a) a high qualityimage illustrated in FIG. 17. This high quality image is the imagegenerated on the basis of each consecutive photographing image in stepS103.

A clear fingerprint (feature point) 221 is photographed in this highquality image. Highly accurate determination processing for identity canbe performed by verifying the high quality image against (b) aregistered image registered in the memory 230 in advance.

The processing according to the present disclosure thus performs themovement vector detection processing in microlens photographing imageunits, and can thereby omit the whole image generation processing step(step S12 in FIG. 2) in consecutive photographing image units, so thatefficient processing is realized.

In addition, more accurate movement vector calculation can be performedby calculating one final movement vector from a plurality of movementvectors corresponding to an identical feature point detected fromduplicate image regions, as described with reference to FIG. 9 and FIG.10.

[5. Examples of Configuration and Processing Sequences of ImageProcessing Devices According to Present Disclosure]

Next, referring to FIG. 18 and following figures, description will bemade of examples of configuration and processing sequences of imageprocessing devices according to the present disclosure. Incidentally, inthe following, the following three image processing devices will bedescribed in order as image processing devices using the movement vectorcalculation processing according to the present disclosure.

(First Embodiment) an image processing device that performs fingerprintauthentication and registration processing

(Second Embodiment) an image processing device that detects an operation(a flick, a swipe, or the like) of a finger on a display screen

(Third Embodiment) an image processing device that performs moving imagecompression processing

(First Embodiment) an Image Processing Device that Performs FingerprintAuthentication and Registration Processing

First, an example of configuration and a processing sequence of an imageprocessing device that performs fingerprint authentication processingand fingerprint registration processing will be described as a firstembodiment.

FIG. 18 is a diagram illustrating an example of configuration of animage processing device 300 according to the present first embodiment.

As illustrated in FIG. 18, the image processing device 300 includes animaging section 301, a movement vector calculating section 302, an imagesynthesizing section 303, an image verifying/registering section 304,and a memory 305. Incidentally, the block diagram illustrated in FIG. 18is a block diagram illustrating only main constituent elements necessaryfor the processing according to the present disclosure. The imageprocessing device 300 is not limited to the constituent elementsillustrated in FIG. 18 but includes, for example, constituent elementssuch as a control section, a storage section, an input section, and anoutput section.

Incidentally, the imaging section 301 may be of a configuration separatefrom the image processing device. In this case, the image processingdevice 300 is supplied with a captured image of the imaging section 301via the input section not illustrated and performs processing.

Each constituent section will be described.

The imaging section 301 is an imaging section having a microlens array(MLA) described earlier with reference to FIG. 1. A photographing imageis photographed via a plurality of microlenses. For example, a movingimage of a finger of a user when the finger is slid on the screen of thesmartphone is photographed. Images of respective frames included in themoving image each include a large number of microlens photographingimages. Each of the microlens photographing images is an image invertedvertically and horizontally and is an image partly including regionsduplicated in other microlens photographing images.

The moving image as the photographing images of the imaging section 301is input to the movement vector calculating section 302.

The movement vector calculating section 302 performs processing ofcalculating a movement vector (optical flow) in each frame image of themoving image as the photographing images of the imaging section 301.

The movement vector calculating section 302 performs the movement vector(optical flow) calculation processing using a large number of microlensphotographing images as they are.

The movement vector calculating section 302 detects a feature point froma microlens photographing image, and according to the position of thedetected feature point, the movement vector calculating section 302performs any one of the three types of movement vector calculationprocessing described earlier with reference to FIG. 9 and FIG. 10, thatis, the following three kinds of processing.

(1) movement vector calculation processing corresponding to a featurepoint in a single image region

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions

The movement vector (optical flow) detected by the movement vectorcalculating section 302 is output to the image synthesizing section 303.

The image synthesizing section 303 aligns consecutive photographingimages included in the moving image photographed by the imaging section301, by using the movement vector (optical flow) detected by themovement vector calculating section 302, and generates a high qualityimage. The high quality image having clear fingerprint information ofthe finger and the like is generated as a result of this processing.

Specifically, for example, image quality improvement processing isperformed by processing such as alpha blending of corresponding pixelsin different photographing timings. One whole image of a high quality isthereafter generated by combining the microlens photographing imagesimproved in image quality with one another. Here, it is necessary toperform vertical and horizontal inversion processing of the individualmicrolens photographing images, an image cutout that detects and deletesoverlap regions, and image combination processing after the cutout.However, this processing is only one piece of whole image generationprocessing in the processing according to the present disclosure.

The high quality image generated by the image synthesizing section 303is output to the image verifying/registering section 304.

The image verifying/registering section 304 is supplied with the highquality image of the finger from the image synthesizing section 303 andperforms image verification processing or registration processing.

At the time of performing the fingerprint authentication processing, theprocessing of verifying the high quality image of the finger which highquality image is input from the image synthesizing section 303 against aregistered image already stored in a memory 315 is performed. In a casewhere the verification is established, the processing proceeds to nextprocessing. In a case of a verification error, a message indicating anauthentication error is output.

At the time of fingerprint registration processing, on the other hand,the high quality image of the finger which high quality image is inputfrom the image synthesizing section 303 is stored as registeredinformation in the memory 315.

A processing sequence of the image processing device 300 illustrated inFIG. 18 will be described with reference to a flowchart illustrated inFIG. 19.

Incidentally, processing according to the flowchart illustrated in FIG.19 is, for example, processing executable according to a program storedin the storage section of the image processing device, and can beperformed under control of a control section (data processing section)including a CPU having a program executing function or the like.

The processing of each step of the flow illustrated in FIG. 19 will bedescribed in order in the following.

(Step S201)

First, in step S201, consecutive photographing images, that is, a movingimage, is photographed.

This processing is performed by the imaging section 301 of the imageprocessing device 300 illustrated in FIG. 18. The imaging section 301photographs the photographing images via a plurality of microlenses. Forexample, a moving image of a finger of a user when the finger is slid onthe screen of the smartphone is photographed.

The moving image as the photographing images of the imaging section 301is input to the movement vector calculating section 302.

(Step S202)

Next, in step S202, the movement vector calculation processing inmicrolens photographing image units is performed. This processing isperformed by the movement vector calculating section 302. The movementvector calculating section 302 performs processing of calculating amovement vector (optical flow) in each frame image of the moving imageas the photographing images of the imaging section 301. The movementvector calculating section 302 performs the movement vector (opticalflow) calculation processing using a large number of microlensphotographing images as they are.

The movement vector calculating section 302 detects a feature point froma microlens photographing image, and according to the position of thedetected feature point, the movement vector calculating section 302performs any one of the three types of movement vector calculationprocessing described earlier with reference to FIG. 9 and FIG. 10, thatis, the following three kinds of processing.

(1) movement vector calculation processing corresponding to a featurepoint in a single image region

(2) movement vector calculation processing corresponding to a featurepoint in two-image duplicate regions

(3) movement vector calculation processing corresponding to a featurepoint in four-image duplicate regions

The movement vector (optical flow) detected by the movement vectorcalculating section 302 is output to the image synthesizing section 303.

(Step S203)

Next, in step S203, alignment processing using movement vectors inmicrolens photographing image units is performed. This processing isperformed by the image synthesizing section 303.

The image synthesizing section 303 aligns consecutive photographingimages included in the moving image photographed by the imaging section301, by using the movement vector (optical flow) detected by themovement vector calculating section 302, and generates a high qualityimage. The high quality image having clear fingerprint information ofthe finger and the like is generated as a result of this processing.

(Step S204)

Further, in step S204, the image synthesizing section 303 generates onewhole image of a high quality by combining the microlens photographingimages improved in image quality with one another. Performed here arethe vertical and horizontal inversion processing of the individualmicrolens photographing images, an image cutout that detects and deletesoverlap regions, and image combination processing after the cutout. Thehigh quality image generated by the image synthesizing section 303 isoutput to the image verifying/registering section 304.

(Steps S205 to S211)

The next processing of steps S205 to S211 is processing performed by theimage verifying/registering section 304. The image verifying/registeringsection 304 is supplied with the high quality image of the finger fromthe image synthesizing section 303 and performs image verificationprocessing or registration processing.

In step S205, whether processing to be performed is the fingerprintauthentication processing or the fingerprint registration processing isdetermined. This determination can be made according to a setting of anapplication at the time of processing disclosure or the like.

In a case where the processing to be performed is the fingerprintauthentication processing, the processing of steps S206 to S209 isperformed.

In a case where the processing to be performed is the fingerprintregistration processing, on the other hand, the processing of step S211is performed.

In the case where the processing to be performed is the fingerprintauthentication processing, processing of verifying the high qualityimage of the finger which high quality image is input from the imagesynthesizing section 303 against a registered image already stored inthe memory 315 is performed in step S206. In a case where theverification is established (step S207=Yes), a verificationestablishment message is output in step S208, and the processingproceeds to next processing. In a case where the verification is notestablished (step S207=No), on the other hand, a verification errormessage is output.

In a case where it is determined in step S205 that the fingerprintregistration processing is to be performed, on the other hand, theprocessing proceeds to step S211, where the high quality image of thefinger which high quality image is input from the image synthesizingsection 303 is stored as registered information in the memory 315.

(Second Embodiment) an Image Processing Device that Detects an Operation(a Flick, a Swipe, or the Like) of a Finger on a Display Screen

Next, an example of configuration and a processing sequence of an imageprocessing device that detects an operation (a flick, a swipe, or thelike) of a finger on a display screen will be described as a secondembodiment.

FIG. 20 is a diagram illustrating an example of configuration of animage processing device 310 according to the present second embodiment.

As illustrated in FIG. 20, the image processing device 310 includes animaging section 311, a movement vector calculating section 312, a movingdirection determining section 313, a processing executing section 314,and the memory 315. Incidentally, the block diagram illustrated in FIG.20 is a block diagram illustrating only main constituent elementsnecessary for the processing according to the present disclosure. Theimage processing device 310 is not limited to the constituent elementsillustrated in FIG. 20 but includes, for example, constituent elementssuch as a control section, a storage section, an input section, and anoutput section.

Incidentally, the imaging section 311 may be of a configuration separatefrom the image processing device. In this case, the image processingdevice 310 is supplied with a captured image of the imaging section 311via the input section not illustrated and performs processing.

Each constituent section will be described.

The imaging section 311 and the movement vector calculating section 312perform processing similar to that of the imaging section 301 and themovement vector calculating section 302 described earlier with referenceto FIG. 18. That is, the imaging section 311 is an imaging sectionhaving a microlens array (MLA). The movement vector calculating section312 performs the movement vector (optical flow) calculation processingusing a large number of microlens photographing images as they are.

A movement vector (optical flow) detected by the movement vectorcalculating section 312 is input to the moving direction determiningsection 313.

The moving direction determining section 313 determines a movementdirection of a finger of a user by using the movement vector (opticalflow) detected by the movement vector calculating section 312.Specifically, the movement direction of the finger in a flick operationor a swipe operation by the finger of the user is determined.

Incidentally, the flick operation is an operation of moving the fingerso as to flick the finger on the screen of a smartphone or the like, andthe swipe operation is processing of sliding the finger while holdingthe finger in contact with the screen.

The moving direction determining section 313 determines the movementdirection of the finger of the user by using the movement vector(optical flow) detected by the movement vector calculating section 312.The movement vector calculating section 312 calculates a large number ofmovement vectors in image region units of the microlens photographingimages. The moving direction determining section 313 determines themovement direction of the finger of the user by, for example,calculating an average value of the large number of movement vectors orthe like.

The moving direction information of the finger of the user which movingdirection information is determined by the moving direction determiningsection 313 is output to the processing executing section 314.

The processing executing section 314 performs processing correspondingto the moving direction of the finger of the user which moving directionis determined by the moving direction determining section 313. Theprocessing is processing corresponding to the flick or swipe operationof the user, such as processing of updating the display screen of thesmartphone. Data stored in the memory 315 and an application, forexample, are used to perform the processing.

A processing sequence of the image processing device 310 illustrated inthis FIG. 20 will be described with reference to a flowchart illustratedin FIG. 21.

Incidentally, processing according to the flowchart illustrated in FIG.21 is, for example, processing executable according to a program storedin the storage section of the image processing device, and can beperformed under control of a control section (data processing section)including a CPU having a program executing function or the like.

The processing of each step of the flow illustrated in FIG. 21 will bedescribed in order in the following.

(Steps S221 and S222)

The processing of steps S221 and S222 is processing similar to theprocessing of steps S201 and S202 of the flow described earlier withreference to FIG. 19.

That is, in step S221, consecutive photographing images, that is, amoving image, is photographed.

In this processing, the imaging section 311 of the image processingdevice 310 illustrated in FIG. 20 photographs the photographing imagesvia a plurality of microlenses. For example, a moving image of a fingerof a user when the finger is slid on the screen of the smartphone isphotographed.

The moving image as the photographing images of the imaging section 311is input to the movement vector calculating section 312.

The movement vector calculating section 312 performs the movement vectorcalculation processing in microlens photographing image units in stepS222. The movement vector calculating section 312 performs processing ofcalculating a movement vector (optical flow) in each frame image of themoving image as the photographing images of the imaging section 311. Themovement vector calculating section 312 performs the movement vector(optical flow) calculation processing using a large number of microlensphotographing images as they are.

The movement vector (optical flow) detected by the movement vectorcalculating section 312 is output to the moving direction determiningsection 313.

(Step S223)

Next, in step S223, moving direction (flick or swipe indicatingdirection) determination processing using movement vectors in microlensphotographing image units is performed. This processing is performed bythe moving direction determining section 313 illustrated in FIG. 20.

The moving direction determining section 313 determines a movementdirection of the finger of the user by calculating an average value of alarge number of movement vectors in image region units of the microlensphotographing images which movement vectors are detected by the movementvector calculating section 312 or the like.

The moving direction information of the finger of the user which movingdirection information is determined by the moving direction determiningsection 313 is output to the processing executing section 314.

(Step S224)

Next, in step S224, processing based on a moving direction (flick orswipe indicating direction) determination result is performed. Thisprocessing is performed by the processing executing section 314.

The processing executing section 314 performs processing correspondingto the moving direction of the finger of the user which moving directionis determined by the moving direction determining section 313. Theprocessing is, for example, processing corresponding to the flick orswipe operation of the user, such as processing of updating the displayscreen of the smartphone. Data stored in the memory 315 and anapplication, for example, are used to perform the processing.

(Third Embodiment) an Image Processing Device that Performs Moving ImageCompression Processing

An example of configuration and a processing sequence of an imageprocessing device that performs moving image compression processing willnext be described as a third embodiment.

FIG. 22 is a diagram illustrating an example of configuration of animage processing device 320 according to the present third embodiment.

As illustrated in FIG. 22, the image processing device 320 includes animaging section 321, a movement vector calculating section 322, an imagecompression processing executing section 323, a communicating section324, and a memory 325. Incidentally, the block diagram illustrated inFIG. 22 is a block diagram illustrating only main constituent elementsnecessary for the processing according to the present disclosure. Theimage processing device 320 is not limited to the constituent elementsillustrated in FIG. 22 but includes, for example, constituent elementssuch as a control section, a storage section, an input section, and anoutput section.

Incidentally, the imaging section 321 may be of a configuration separatefrom the image processing device. In this case, the image processingdevice 320 is supplied with a captured image of the imaging section 321via the input section not illustrated and performs processing.

Each constituent section will be described.

The imaging section 321 and the movement vector calculating section 322perform processing similar to that of the imaging section 301 and themovement vector calculating section 302 described earlier with referenceto FIG. 18. That is, the imaging section 321 is an imaging sectionhaving a microlens array (MLA). The movement vector calculating section322 performs the movement vector (optical flow) calculation processingusing a large number of microlens photographing images as they are.

A movement vector (optical flow) detected by the movement vectorcalculating section 322 is input to the image compression processingexecuting section 323.

The image compression processing executing section 323 performs movingimage compression processing using the movement vector (optical flow)detected by the movement vector calculating section 322. Specifically,for example, the moving image compression processing conforming to anMPEG compression algorithm is performed.

The MPEG compression processing performs processing of calculatingdifference information between different frames and encoding thedifference information in order to achieve a high compression ratio. Themovement vector is used for this difference calculation. The imagecompression processing executing section 323 performs the differencecalculation using the movement vector (optical flow) detected by themovement vector calculating section 322, thereby performing the MPEGcompression processing.

An image compression result of the image compression processingexecuting section 323 is output to the outside via the communicatingsection 324. Alternatively, the image compression result of the imagecompression processing executing section 323 is stored in the memory325.

Next, a processing sequence of the image processing device 320illustrated in FIG. 22 will be described with reference to a flowchartillustrated in FIG. 23.

Incidentally, processing according to the flowchart illustrated in FIG.23 is, for example, processing executable according to a program storedin the storage section of the image processing device, and can beperformed under control of a control section (data processing section)including a CPU having a program executing function or the like.

The processing of each step of the flow illustrated in FIG. 23 will bedescribed in order in the following.

(Steps S251 and S252)

The processing of steps S251 and S252 is processing similar to theprocessing of steps S201 and S202 of the flow described earlier withreference to FIG. 19.

That is, in step S251, consecutive photographing images, that is, amoving image, is photographed.

In this processing, the imaging section 321 of the image processingdevice 320 illustrated in FIG. 22 photographs the photographing imagesvia a plurality of microlenses. The moving image as the photographingimages of the imaging section 321 is input to the movement vectorcalculating section 322.

The movement vector calculating section 322 performs the movement vectorcalculation processing in microlens photographing image units in stepS252. The movement vector calculating section 322 performs processing ofcalculating a movement vector (optical flow) in each frame image of themoving image as the photographing images of the imaging section 321. Themovement vector calculating section 322 performs the movement vector(optical flow) calculation processing using a large number of microlensphotographing images as they are.

The movement vector (optical flow) detected by the movement vectorcalculating section 322 is output to the image compression processingexecuting section 323.

(Step S253)

Next, in step S253, image compression (MPEG or the like) processingusing movement vectors in microlens photographing image units isperformed. This processing is performed by the image compressionprocessing executing section 323 illustrated in FIG. 22.

The image compression processing executing section 323 performs movingimage compression processing using the movement vector (optical flow)detected by the movement vector calculating section 322. Specifically,for example, the moving image compression processing conforming to theMPEG compression algorithm is performed. The image compressionprocessing executing section 323 performs difference calculation usingthe movement vector (optical flow) detected by the movement vectorcalculating section 322, thereby performing the MPEG compressionprocessing.

(Step S254)

Next, in step S254, an image compression result of the image compressionprocessing executing section 323 is output to the outside via thecommunicating section 324. Alternatively, the image compression resultof the image compression processing executing section 323 is stored inthe memory 325.

[6. Concrete Device Examples Using Movement Vector CalculationProcessing According to Present Disclosure]

Description will next be made of concrete device examples using themovement vector calculation processing according to the presentdisclosure.

Device examples illustrated in FIG. 24 are examples of fingerprintauthenticating devices in smartphones. The figure illustratesconfiguration examples of a fingerprint authenticating device 401 havingan imaging unit formed in a side surface of a smartphone and afingerprint authenticating device 402 having an imaging unit formed in alower portion of a front surface of a smartphone.

Each of the imaging units is an imaging unit having a microlens array(MLA).

A device example illustrated in FIG. 25(b) is an example of afingerprint authenticating device in a smart watch as a watch typeinformation processing device. The figure illustrates a configurationexample of a fingerprint authenticating device 403 having an imagingunit formed in a front surface of the smart watch.

The imaging unit is an imaging unit having a microlens array (MLA).

A device example illustrated in FIG. 25(c) is an example of an imagingdevice 404 as a close-up high-resolution image photographing camera suchas a skin sensor. The imaging unit is an imaging unit having a microlensarray (MLA). After movement vector detection in microlens photographingimage units as described earlier is performed, a high quality image isgenerated by alignment to which a detected movement vector is appliedand image synthesis processing, and is displayed on a display unit of aPC or the like.

FIG. 26 illustrates concrete device configuration examples forprocessing of detecting a movement of a finger of a user such as a flickor swipe operation as described earlier with reference to FIG. 20 andFIG. 21.

A device that performs high-speed and high-accuracy moving directiondetection is implemented by applying the movement vector detectionaccording to the present disclosure to a configuration for performingcharacter input by a flick operation as illustrated in a lower part ofFIG. 26, for example, in a smartphone, a smart watch, or a ring typeterminal.

[7. Example of Hardware Configuration of Image Processing Device]

Next, an example of hardware configuration of the image processingdevice will be described with reference to FIG. 27.

FIG. 27 is a diagram illustrating an example of hardware configurationof the image processing device that performs the processing according tothe present disclosure.

A CPU (Central Processing Unit) 601 functions as a control unit or adata processing unit that performs various kinds of processing accordingto a program stored in a ROM (Read Only Memory) 602 or a storage unit608. For example, processing according to the sequences described in theforegoing embodiments is performed. A RAM (Random Access Memory) 603stores the program executed by the CPU 601, data, and the like. The CPU601, the ROM 602, and the RAM 603 are interconnected by a bus 604.

The CPU 601 is connected to an input-output interface 605 via the bus604. Connected with the input-output interface 605 are an imaging unit621 having a microlens array, an input unit 606 including various kindsof switches capable of user input, a keyboard, a mouse, a microphone,and the like, and an output unit 607 that performs data output to adisplay unit, a speaker, or the like. The CPU 601 performs various kindsof processing in response to commands input from the input unit 606 andoutputs a processing result to the output unit 607, for example.

The storage unit 608 connected to the input-output interface 605, forexample, includes a hard disk or the like. The storage unit 608 storesthe program executed by the CPU 601 and various kinds of data. Acommunicating unit 609 communicates with an external device byfunctioning as a transmitting and receiving unit in Wi-Fi communication,Bluetooth (registered trademark) (BT) communication, and other datacommunication via a network such as the Internet or a local areanetwork.

A drive 610 connected to the input-output interface 605 records or readsdata by driving a removable medium 611 including a magnetic disk, anoptical disk, a magneto-optical disk, a semiconductor memory such as amemory card, or the like.

[8. Summary of Configuration According to Present Disclosure]

Embodiments of the present disclosure have been explained above indetail with reference to particular embodiments. However, it is obviousthat those skilled in the art can make modifications and substitutionsin the embodiments without departing from the spirit of the presentdisclosure. That is, the present invention has been disclosed in anillustrative form and is not to be construed in a limited manner. Inorder to determine the gist of the present disclosure, the section ofclaims is to be considered.

Incidentally, the technology disclosed in the present specification canadopt the following configurations.

(1)

An image processing device including:

a movement vector calculating section configured to receive, as aninput, consecutive photographing images of an imaging section having amicrolens array and calculate a movement vector between the images,

the movement vector calculating section calculating the movement vectorcorresponding to a feature point by using microlens photographing imagesphotographed by respective microlenses included in the microlens array.

(2)

The image processing device according to (1), in which

the movement vector calculating section performs, as differentprocessing,

-   -   processing of calculating the movement vector corresponding to        the feature point within duplicate image regions having the same        image region within a plurality of microlens photographing        images, and    -   processing of calculating the movement vector corresponding to        the feature point within a single image region not having the        same image region within a plurality of microlens photographing        images.        (3)

The image processing device according to (2), in which

when the movement vector calculating section calculates the movementvector corresponding to the feature point within the duplicate imageregions,

the movement vector calculating section calculates an average value of aplurality of movement vectors corresponding to the same feature pointand sets the average value as the movement vector of the feature point.

(4)

The image processing device according to (2) or (3), in which

when the movement vector calculating section calculates the movementvector corresponding to the feature point within the duplicate imageregions,

the movement vector calculating section calculates an average value of aplurality of movement vectors remaining after an outlier (abnormalvalue) is excluded among a plurality of movement vectors correspondingto the same feature point, and sets the average value as the movementvector of the feature point.

(5)

The image processing device according to any one of (1) to (4), in which

the movement vector calculating section distinguishes

-   -   a single image region not having the same image region within a        plurality of microlens photographing images, and    -   duplicate image regions having the same image region within a        plurality of microlens photographing images,

for the duplicate image regions, the movement vector calculating sectionfurther distinguishes

-   -   two-image duplicate regions having two identical image regions,        and    -   four-image duplicate regions having four identical image        regions, and

the movement vector calculating section performs movement vectorcalculation processing different in each region unit.

(6)

The image processing device according to any one of (1) to (5), in which

as movement vector calculation processing corresponding to a featurepoint detected from a microlens photographing image photographed by eachmicrolens in an outermost circumference region of the microlens array,among the microlens photographing images photographed by the respectivemicrolenses included in the microlens array, the movement vectorcalculating section performs exception processing different from themovement vector calculation processing corresponding to feature pointsdetected from other microlens photographing images.

(7)

The image processing device according to any one of (1) to (6), in which

the movement vector calculating section performs different movementvector calculation processing according to whether or not there areduplicate image regions within a plurality of microlens photographingimages, and according to the number of duplications in a case wherethere are duplicate image regions.

(8)

The image processing device according to any one of (1) to (7), in which

the movement vector calculating section performs a feature point searchafter setting a search range in which a position of a microlens includedin the microlens array and a manner of image inversion are taken intoconsideration.

(9)

The image processing device according to (8), in which

the movement vector calculating section performs the feature pointsearch after setting the search range in which image duplicate regionsare further taken into consideration.

(10)

The image processing device according to any one of (1) to (9), furtherincluding:

an image synthesizing section configured to perform synthesis processingof the consecutive photographing images of the imaging section by usingthe movement vector calculated by the movement vector calculatingsection.

(11)

The image processing device according to (10), in which

the image synthesizing section aligns each of consecutive photographingmicrolens photographing images photographed by the respectivemicrolenses included in the microlens array, by using the movementvector calculated by the movement vector calculating section, andperforms synthesis processing of each of the consecutive photographingmicrolens photographing images.

(12)

The image processing device according to (10) or (11), in which

the consecutive photographing images of the imaging section includefinger images including a fingerprint of a person, and

the image processing device further includes an image verifying sectionconfigured to perform processing of verification against a registeredfingerprint image by using a synthetic image generated by the imagesynthesizing section.

(13)

An image processing device for performing fingerprint authenticationprocessing, the image processing device including:

a movement vector calculating section configured to receive, as aninput, consecutive photographing images of a finger photographed by animaging section having a microlens array and calculate a movement vectorbetween the images;

an image synthesizing section configured to generate a high qualityimage by synthesis processing of the consecutive photographing images byusing the movement vector calculated by the movement vector calculatingsection; and

an image verifying section configured to perform processing of verifyinga synthetic image generated by the image synthesizing section against aregistered fingerprint image stored in a storage section in advance.

(14)

The image processing device according to (13), in which

the movement vector calculating section calculates the movement vectorcorresponding to a feature point by using microlens photographing imagesphotographed by respective microlenses included in the microlens array,and

the image synthesizing section aligns each of consecutive photographingmicrolens photographing images photographed by the respectivemicrolenses included in the microlens array, by using the movementvector calculated by the movement vector calculating section, andperforms synthesis processing of each of the consecutive photographingmicrolens photographing images.

(15)

An image processing method performed in an image processing device, theimage processing method including:

by a movement vector calculating section, a movement vector calculatingstep of receiving, as an input, consecutive photographing images of animaging section having a microlens array and calculating a movementvector between the images, in which

the movement vector calculating step includes a step of calculating themovement vector corresponding to a feature point by using microlensphotographing images photographed by respective microlenses included inthe microlens array.

(16)

An image processing method for performing fingerprint authenticationprocessing in an image processing device, the image processing methodincluding:

by a movement vector calculating section, a movement vector calculatingstep of receiving, as an input, consecutive photographing images of afinger photographed by an imaging section having a microlens array andcalculating a movement vector between the images;

by an image synthesizing section, an image synthesizing step ofgenerating a high quality image by synthesis processing of theconsecutive photographing images by using the movement vector calculatedby the movement vector calculating section; and

by an image verifying section, an image verifying step of performingprocessing of verifying a synthetic image generated by the imagesynthesizing section against a registered fingerprint image stored in astorage section in advance.

(17)

A program for making image processing performed in an image processingdevice, the program including:

making a movement vector calculating section perform a movement vectorcalculating step of receiving, as an input, consecutive photographingimages of an imaging section having a microlens array and calculating amovement vector between the images, in which

in the movement vector calculating step, processing of calculating themovement vector corresponding to a feature point by using microlensphotographing images photographed by respective microlenses included inthe microlens array is made to be performed.

(18)

A program for making fingerprint authentication processing performed inan image processing device, the program including:

making a movement vector calculating section perform a movement vectorcalculating step of receiving, as an input, consecutive photographingimages of a finger photographed by an imaging section having a microlensarray and calculating a movement vector between the images;

making an image synthesizing section perform an image synthesizing stepof generating a high quality image by synthesis processing of theconsecutive photographing images by using the movement vector calculatedby the movement vector calculating section; and

making an image verifying section perform an image verifying step ofperforming processing of verifying a synthetic image generated by theimage synthesizing section against a registered fingerprint image storedin a storage section in advance.

In addition, a series of processing described in the specification canbe performed by hardware, software, or a composite configuration ofhardware and software. In a case where the processing is performed bysoftware, the processing can be performed after a program in which aprocessing sequence is recorded is installed in a memory within acomputer incorporated in dedicated hardware, or the processing can beperformed after the program is installed in a general-purpose computercapable of performing various kinds of processing. For example, theprogram can be recorded on a recording medium in advance. In addition tobeing installed from a recording medium to a computer, the program canbe received via a network such as a LAN (Local Area Network) or theInternet, and installed on a recording medium such as a built-in harddisk.

Incidentally, the various kinds of processing described in thespecification may be not only performed in time series according to thedescription but also performed in parallel or individually according tothe processing power of a device performing the processing or asrequired. In addition, a system in the present specification is alogical set configuration of a plurality of devices and is not limitedto a system in which the devices of respective configurations arelocated within the same casing.

INDUSTRIAL APPLICABILITY

As described above, according to a configuration based on one embodimentof the present disclosure, a device and a method are implemented whichperform movement vector calculation processing from photographing imagesfor which a microlens array is used without generating whole images.

Specifically, for example, consecutive photographing images photographedby an imaging section having a microlens array are input, and a movementvector between the images is calculated. A movement vector calculatingsection calculates the movement vector corresponding to a feature pointby using microlens photographing images photographed by respectivemicrolenses. When the movement vector corresponding to the feature pointwithin duplicate image regions having the same image region within aplurality of microlens photographing images is calculated, an averagevalue of a plurality of movement vectors corresponding to the samefeature point is calculated and is set as the movement vector of thefeature point. Alternatively, an average value of plural movementvectors remaining after an outlier (abnormal value) is excluded iscalculated and is set as the movement vector of the feature point.

A device and a method that perform movement vector calculationprocessing from photographing images for which a microlens array is usedwithout generating whole images are implemented by these pieces ofprocessing.

REFERENCE SIGNS LIST

-   -   10 Microlens array (MLA)    -   11 Microlens    -   12 Image sensor (imaging element)    -   101 Microlens photographing image    -   300 Image processing device    -   301 Imaging section    -   302 Movement vector calculating section    -   303 Image synthesizing section    -   304 Image verifying/registering section    -   305 Memory    -   310 Image processing device    -   311 Imaging section    -   312 Movement vector calculating section    -   313 Moving direction determining section    -   314 Processing executing section    -   315 Memory    -   320 Image processing device    -   321 Imaging section    -   322 Movement vector calculating section    -   323 Image compression processing executing section    -   324 Communicating section    -   325 Memory    -   401, 402, 403 Fingerprint authenticating device    -   404 Imaging device    -   601 CPU    -   602 ROM    -   603 RAM    -   604 Bus    -   605 Input-output interface    -   606 Input unit    -   607 Output unit    -   608 Storage unit    -   609 Communicating unit    -   610 Drive    -   611 Removable medium    -   621 Imaging unit

1. An image processing device comprising: a movement vector calculatingsection configured to receive, as an input, consecutive photographingimages of an imaging section having a microlens array and calculate amovement vector between the images, the movement vector calculatingsection calculating the movement vector corresponding to a feature pointby using microlens photographing images photographed by respectivemicrolenses included in the microlens array.
 2. The image processingdevice according to claim 1, wherein the movement vector calculatingsection performs, as different processing, processing of calculating themovement vector corresponding to the feature point within duplicateimage regions having a same image region within a plurality of microlensphotographing images, and processing of calculating the movement vectorcorresponding to the feature point within a single image region nothaving a same image region within a plurality of microlens photographingimages.
 3. The image processing device according to claim 2, whereinwhen the movement vector calculating section calculates the movementvector corresponding to the feature point within the duplicate imageregions, the movement vector calculating section calculates an averagevalue of a plurality of movement vectors corresponding to a same featurepoint and sets the average value as the movement vector of the featurepoint.
 4. The image processing device according to claim 2, wherein whenthe movement vector calculating section calculates the movement vectorcorresponding to the feature point within the duplicate image regions,the movement vector calculating section calculates an average value of aplurality of movement vectors remaining after an outlier (abnormalvalue) is excluded among a plurality of movement vectors correspondingto a same feature point, and sets the average value as the movementvector of the feature point.
 5. The image processing device according toclaim 1, wherein the movement vector calculating section distinguishes asingle image region not having a same image region within a plurality ofmicrolens photographing images, and duplicate image regions having asame image region within a plurality of microlens photographing images,for the duplicate image regions, the movement vector calculating sectionfurther distinguishes two-image duplicate regions having two identicalimage regions, and four-image duplicate regions having four identicalimage regions, and the movement vector calculating section performsmovement vector calculation processing different in each region unit. 6.The image processing device according to claim 1, wherein as movementvector calculation processing corresponding to a feature point detectedfrom a microlens photographing image photographed by each microlens inan outermost circumference region of the microlens array, among themicrolens photographing images photographed by the respectivemicrolenses included in the microlens array, the movement vectorcalculating section performs exception processing different from themovement vector calculation processing corresponding to feature pointsdetected from other microlens photographing images.
 7. The imageprocessing device according to claim 1, wherein the movement vectorcalculating section performs different movement vector calculationprocessing according to whether or not there are duplicate image regionswithin a plurality of microlens photographing images and according tothe number of duplications in a case where there are duplicate imageregions.
 8. The image processing device according to claim 1, whereinthe movement vector calculating section performs a feature point searchafter setting a search range in which a position of a microlens includedin the microlens array and a manner of image inversion are taken intoconsideration.
 9. The image processing device according to claim 8,wherein the movement vector calculating section performs the featurepoint search after setting the search range in which image duplicateregions are further taken into consideration.
 10. The image processingdevice according to claim 1, further comprising: an image synthesizingsection configured to perform synthesis processing of the consecutivephotographing images of the imaging section by using the movement vectorcalculated by the movement vector calculating section.
 11. The imageprocessing device according to claim 10, wherein the image synthesizingsection aligns each of consecutive photographing microlens photographingimages photographed by the respective microlenses included in themicrolens array, by using the movement vector calculated by the movementvector calculating section, and performs synthesis processing of each ofthe consecutive photographing microlens photographing images.
 12. Theimage processing device according to claim 10, wherein the consecutivephotographing images of the imaging section include finger imagesincluding a fingerprint of a person, and the image processing devicefurther includes an image verifying section configured to performprocessing of verification against a registered fingerprint image byusing a synthetic image generated by the image synthesizing section. 13.An image processing device for performing fingerprint authenticationprocessing, the image processing device comprising: a movement vectorcalculating section configured to receive, as an input, consecutivephotographing images of a finger photographed by an imaging sectionhaving a microlens array and calculate a movement vector between theimages; an image synthesizing section configured to generate a highquality image by synthesis processing of the consecutive photographingimages by using the movement vector calculated by the movement vectorcalculating section; and an image verifying section configured toperform processing of verifying a synthetic image generated by the imagesynthesizing section against a registered fingerprint image stored in astorage section in advance.
 14. The image processing device according toclaim 13, wherein the movement vector calculating section calculates themovement vector corresponding to a feature point by using microlensphotographing images photographed by respective microlenses included inthe microlens array, and the image synthesizing section aligns each ofconsecutive photographing microlens photographing images photographed bythe respective microlenses included in the microlens array, by using themovement vector calculated by the movement vector calculating section,and performs synthesis processing of each of the consecutivephotographing microlens photographing images.
 15. An image processingmethod performed in an image processing device, the image processingmethod comprising: by a movement vector calculating section, a movementvector calculating step of receiving, as an input, consecutivephotographing images of an imaging section having a microlens array andcalculating a movement vector between the images, wherein the movementvector calculating step includes a step of calculating the movementvector corresponding to a feature point by using microlens photographingimages photographed by respective microlenses included in the microlensarray.
 16. An image processing method for performing fingerprintauthentication processing in an image processing device, the imageprocessing method comprising: by a movement vector calculating section,a movement vector calculating step of receiving, as an input,consecutive photographing images of a finger photographed by an imagingsection having a microlens array and calculating a movement vectorbetween the images; by an image synthesizing section, an imagesynthesizing step of generating a high quality image by synthesisprocessing of the consecutive photographing images by using the movementvector calculated by the movement vector calculating section; and by animage verifying section, an image verifying step of performingprocessing of verifying a synthetic image generated by the imagesynthesizing section against a registered fingerprint image stored in astorage section in advance.
 17. A program for making image processingperformed in an image processing device, the program comprising: makinga movement vector calculating section perform a movement vectorcalculating step of receiving, as an input, consecutive photographingimages of an imaging section having a microlens array and calculating amovement vector between the images, wherein in the movement vectorcalculating step, processing of calculating the movement vectorcorresponding to a feature point by using microlens photographing imagesphotographed by respective microlenses included in the microlens arrayis made to be performed.
 18. A program for making fingerprintauthentication processing performed in an image processing device, theprogram comprising: making a movement vector calculating section performa movement vector calculating step of receiving, as an input,consecutive photographing images of a finger photographed by an imagingsection having a microlens array and calculating a movement vectorbetween the images; making an image synthesizing section perform animage synthesizing step of generating a high quality image by synthesisprocessing of the consecutive photographing images by using the movementvector calculated by the movement vector calculating section; and makingan image verifying section perform an image verifying step of performingprocessing of verifying a synthetic image generated by the imagesynthesizing section against a registered fingerprint image stored in astorage section in advance.